An MCMC Approach to Solving Hybrid Factored MDPs
نویسندگان
چکیده
Hybrid approximate linear programming (HALP) has recently emerged as a promising framework for solving large factored Markov decision processes (MDPs) with discrete and continuous state and action variables. Our work addresses its major computational bottleneck – constraint satisfaction in large structured domains of discrete and continuous variables. We analyze this problem and propose a novel Markov chain Monte Carlo (MCMC) method for finding the most violated constraint of a relaxed HALP. This method does not require the discretization of continuous variables, searches the space of constraints intelligently based on the structure of factored MDPs, and its space complexity is linear in the number of variables. We test the method on a set of large control problems and demonstrate improvements over alternative approaches.
منابع مشابه
Approximate Linear Programming for Solving Hybrid Factored MDPs
Hybrid approximate linear programming (HALP) has recently emerged as a promising approach to solving large factored Markov decision processes (MDPs) with discrete and continuous state and action variables. Its central idea is to reformulate initially intractable problem of computing the optimal value function as its linear programming approximation. In this work, we present the HALP framework a...
متن کاملOn the Smoothness of Linear Value Function Approximations
Markov decision processes (MDPs) with discrete and continuous state and action components can be solved efficiently by hybrid approximate linear programming (HALP). The main idea of the approach is to approximate the optimal value function by a set of basis functions and optimize their weights by linear programming. It is known that the solution to this convex optimization problem minimizes the...
متن کاملOverview of Linear Program Approximations for Factored Continuous and Hybrid-State Markov Decision Processes
Approximate linear programming (ALP) is as one of the most promising methods for solving complex factored MDPs. The method was applied first to tackle problems with discrete state variables. More recently the ALP methods that can solve MDPs with continuous and hybrid (both continuous and discrete) variables have emerged. This paper briefly reviews the work on ALP methods for such problems.
متن کاملSolving Factored MDPs with Continuous and Discrete Variables
Although many real-world stochastic planning problems are more naturally formulated by hybrid models with both discrete and continuous variables, current state-of-the-art methods cannot adequately address these problems. We present the first framework that can exploit problem structure for modeling and solving hybrid problems efficiently. We formulate these problems as hybrid Markov decision pr...
متن کاملSolving Factored MDPs with Large Action Space Using Algebraic Decision Diagrams
We describe an algorithm for solving MDPs with large state and action spaces, represented as factored MDPs with factored action spaces. Classical algorithms for solving MDPs are not effective since they require enumerating all the states and actions. As such, model minimization techniques have been proposed, and specifically, we extend the previous work on model minimization algorithm for MDPs ...
متن کامل